00010000:00010000:1.0F:1769076011.823942:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076012.144004:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076012.144013:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076012.144019:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076012.144034:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1694 00000100:00080000:1.0:1769076012.144076:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076012.144157:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076012.144205:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713e300 x1855008961330432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076012.144269:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076012.144275:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076012.144279:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076012.144286:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1694 00000100:00080000:1.0:1769076012.144293:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076012.144300:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076012.144313:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076012.144334:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f100 x1855008961330560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076012.144352:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076012.144357:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076012.144360:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076012.144366:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1694 00000100:00080000:1.0:1769076012.144372:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076012.144379:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076012.144386:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0F:1769076012.144429:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961330432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076012.144434:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ea00 x1855008961330688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076012.144522:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076012.144580:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076012/real 1769076012] req@ffff895e4713e300 x1855008961330432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076012.144611:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713e300 x1855008961330432/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076012.144627:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076012.144633:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076012.144638:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961330432:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076012.144651:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961330560:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076012.144688:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076012.144715:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076012/real 1769076012] req@ffff895e4713f100 x1855008961330560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076012.144738:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f100 x1855008961330560/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076012.144794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076012.144800:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076012.145648:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961330560:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076012.145689:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961330688:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076012.145735:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076012.145795:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076012/real 1769076012] req@ffff895e4713ea00 x1855008961330688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076012.145843:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ea00 x1855008961330688/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076012.145858:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076012.145867:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076012.145874:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961330688:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769076012.847954:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076013.871816:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076014.896041:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076015.919950:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076016.943951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1769076016.980703:0:20621:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-sec test 13: test nids ========================= 05:00:16 (1769076016) 00000100:00080000:1.0:1769076017.263979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076017.263994:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076017.264017:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076017.264033:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1699 00000100:00080000:1.0:1769076017.264050:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076017.264062:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076017.264116:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d500 x1855008961330816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076017.264158:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076017.264162:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076017.264181:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076017.264186:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1699 00000100:00080000:1.0:1769076017.264190:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076017.264194:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076017.264214:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076017.264232:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961330816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076017.264233:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ce00 x1855008961330944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076017.264255:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076017.264258:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076017.264277:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076017.264282:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1699 00000100:00080000:1.0:1769076017.264287:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076017.264292:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076017.264299:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076017.264313:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f480 x1855008961331072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1769076017.264329:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076017.264384:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076017/real 1769076017] req@ffff895e4713d500 x1855008961330816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076017.264413:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d500 x1855008961330816/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076017.264425:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076017.264429:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076017.264434:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961330816:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076017.264473:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961330944:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076017.264508:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076017.264527:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076017/real 1769076017] req@ffff895e4713ce00 x1855008961330944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076017.264574:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ce00 x1855008961330944/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076017.264586:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076017.264591:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076017.264597:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961330944:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076017.264608:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331072:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076017.264643:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076017.264661:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076017/real 1769076017] req@ffff895e4713f480 x1855008961331072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076017.264679:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f480 x1855008961331072/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076017.264691:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076017.264695:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076017.264701:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331072:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769076017.967973:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076018.992005:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076020.015954:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076021.040028:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076022.063989:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076022.383940:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076022.383975:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076022.383983:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076022.384000:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1704 00000100:00080000:3.0:1769076022.384018:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076022.384029:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076022.384090:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5500 x1855008961331200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076022.384136:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076022.384144:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076022.384147:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076022.384157:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1704 00000100:00080000:3.0:1769076022.384167:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076022.384178:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076022.384196:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076022.384252:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7480 x1855008961331328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076022.384281:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076022.384285:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076022.384288:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0F:1769076022.384298:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961331200:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076022.384299:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1704 00000100:00080000:3.0:1769076022.384323:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076022.384345:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076022.384363:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076022.384395:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4e00 x1855008961331456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076022.384469:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076022.384573:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076022/real 1769076022] req@ffff895e453d5500 x1855008961331200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076022.384612:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5500 x1855008961331200/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076022.384630:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076022.384638:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076022.384646:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961331200:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076022.384685:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331328:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076022.384814:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076022.384847:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076022/real 1769076022] req@ffff895e453d7480 x1855008961331328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076022.384901:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7480 x1855008961331328/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076022.384921:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076022.384930:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076022.384939:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331328:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076022.384953:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331456:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076022.384996:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076022.385017:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076022/real 1769076022] req@ffff895e453d4e00 x1855008961331456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076022.385065:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4e00 x1855008961331456/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076022.385082:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076022.385088:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076022.385094:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331456:192.168.204.117@tcp:38:kworker.0 00010000:00010000:3.0:1769076023.088020:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076024.112039:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076025.136017:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076026.159951:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076027.183998:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1769076027.503950:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076027.503963:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1769076027.503969:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076027.503984:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1710 00000100:00080000:0.0:1769076027.503999:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076027.504009:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:0.0:1769076027.504052:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ea00 x1855008961331584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1769076027.504094:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076027.504099:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1769076027.504101:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076027.504106:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1710 00000100:00080000:0.0:1769076027.504110:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076027.504115:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076027.504127:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076027.504144:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383dc00 x1855008961331712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076027.504173:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961331584:192.168.204.117@tcp:250:kworker.0 00000100:00080000:0.0:1769076027.504212:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1769076027.504223:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1769076027.504227:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1769076027.504238:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1710 00000100:00080000:0.0:1769076027.504248:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1769076027.504257:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:0.0:1769076027.504269:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076027.504303:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4383ca80 x1855008961331840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076027.504408:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076027.504467:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331840:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769076028.208040:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1769076028.528420:0:2393:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.117:988 00000400:00000100:1.0:1769076028.528534:0:2393:0:(acceptor.c:115:lnet_connect_console_error()) Connection to 192.168.204.117@tcp at host 192.168.204.117 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1769076028.528567:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076028.528635:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000800:00000100:1.0:1769076028.528670:0:2393:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.204.117@tcp 00000100:00100000:3.0:1769076028.528885:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076027/real 1769076028] req@ffff895e4383dc00 x1855008961331712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076028.528963:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383dc00 x1855008961331712/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076028.528990:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076028.529002:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076028.529011:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331712:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076028.529028:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076027/real 1769076028] req@ffff895e4383ca80 x1855008961331840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076028.529054:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ca80 x1855008961331840/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076028.529070:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076028.529076:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076028.529083:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961331840:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076028.529098:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076027/real 1769076028] req@ffff895e4383ea00 x1855008961331584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076028.529121:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4383ea00 x1855008961331584/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076028.529178:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076028.529187:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076028.529195:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4383ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961331584:192.168.204.117@tcp:250:kworker.0 00010000:00010000:3.0:1769076029.232000:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076030.256038:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0F:1769076031.279953:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076032.303900:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076032.623915:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076032.623928:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076032.623933:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076032.623946:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1715 00000100:00080000:1.0:1769076032.623959:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076032.623968:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076032.624028:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713ca80 x1855008961331968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076032.624069:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076032.624074:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076032.624076:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076032.624082:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1715 00000100:00080000:1.0:1769076032.624089:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076032.624098:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076032.624115:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076032.624143:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961331968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076032.624145:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713f800 x1855008961332096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076032.624199:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076032.624207:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076032.624211:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076032.624219:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1715 00000100:00080000:1.0:1769076032.624227:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076032.624235:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076032.624245:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1769076032.624274:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:1.0:1769076032.624277:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713dc00 x1855008961332224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1769076032.624354:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076032/real 1769076032] req@ffff895e4713ca80 x1855008961331968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076032.624403:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713ca80 x1855008961331968/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076032.624434:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076032.624442:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076032.624450:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961331968:192.168.204.117@tcp:250:kworker.0 00000100:00100000:3.0:1769076032.624516:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332096:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076032.624566:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076032.624594:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076032/real 1769076032] req@ffff895e4713f800 x1855008961332096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076032.624621:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713f800 x1855008961332096/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076032.624639:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076032.624663:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076032.624671:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332096:192.168.204.117@tcp:38:kworker.0 00000100:00100000:3.0:1769076032.624685:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332224:192.168.204.117@tcp:38:kworker.0 00000800:00000100:3.0:1769076032.625599:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:3.0:1769076032.625634:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076032/real 1769076032] req@ffff895e4713dc00 x1855008961332224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1769076032.625669:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713dc00 x1855008961332224/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1769076032.625730:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1769076032.625742:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:3.0:1769076032.625789:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332224:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769076033.327959:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769076034.352000:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076035.375977:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076036.399971:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769076037.424033:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076037.744266:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076037.744278:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076037.744284:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076037.744298:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1720 00000100:00080000:3.0:1769076037.744312:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076037.744324:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076037.744373:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d7800 x1855008961332352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076037.744427:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076037.744433:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076037.744437:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076037.744456:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1720 00000100:00080000:3.0:1769076037.744465:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076037.744473:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076037.744488:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076037.744510:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961332480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1769076037.744531:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961332352:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076037.744537:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076037.744551:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076037.744557:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076037.744568:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1720 00000100:00080000:3.0:1769076037.744605:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076037.744637:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076037.744649:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076037.744682:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961332608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076037.744683:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076037.744825:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076037/real 1769076037] req@ffff895e453d7800 x1855008961332352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076037.744868:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d7800 x1855008961332352/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076037.744920:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076037.744930:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076037.744939:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961332352:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076037.744995:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332480:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076037.745049:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076037.745106:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076037/real 1769076037] req@ffff895e453d5c00 x1855008961332480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076037.745150:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961332480/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076037.745168:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076037.745176:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076037.745182:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332480:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076037.745195:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332608:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076037.745236:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076037.745262:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076037/real 1769076037] req@ffff895e453d4a80 x1855008961332608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076037.745286:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961332608/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076037.745301:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076037.745308:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076037.745315:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332608:192.168.204.117@tcp:38:kworker.0 00010000:00010000:3.0:1769076038.447897:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076039.472026:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769076040.495956:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076041.519943:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076042.543965:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1769076042.863897:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076042.863909:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1769076042.863914:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076042.863926:0:18188:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1725 00000100:00080000:1.0:1769076042.863940:0:18188:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076042.863949:0:18188:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:1.0:1769076042.863992:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713d180 x1855008961332736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076042.864027:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076042.864031:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1769076042.864034:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076042.864038:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1725 00000100:00080000:1.0:1769076042.864041:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076042.864045:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076042.864055:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1769076042.864068:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713c380 x1855008961332864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1769076042.864085:0:18188:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1769076042.864091:0:18188:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1769076042.864094:0:18188:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1769076042.864103:0:18188:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1725 00000100:00080000:1.0:1769076042.864110:0:18188:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1769076042.864117:0:18188:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:1.0:1769076042.864126:0:18188:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1769076042.864173:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961332736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:1.0:1769076042.864174:0:18188:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e4713df80 x1855008961332992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076042.864324:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076042.864435:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076042/real 1769076042] req@ffff895e4713d180 x1855008961332736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076042.864471:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713d180 x1855008961332736/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076042.864575:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076042.864587:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076042.864614:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961332736:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076042.864630:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332864:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076042.864681:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076042.864706:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076042/real 1769076042] req@ffff895e4713c380 x1855008961332864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076042.864729:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713c380 x1855008961332864/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076042.864794:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076042.864802:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076042.864809:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332864:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076042.864870:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332992:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076042.864919:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076042.864943:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076042/real 1769076042] req@ffff895e4713df80 x1855008961332992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076042.864963:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e4713df80 x1855008961332992/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076042.864977:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076042.864982:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076042.864987:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e4713df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961332992:192.168.204.117@tcp:38:kworker.0 00010000:00010000:2.0:1769076043.568007:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1769076044.591942:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1769076045.615966:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769076046.640000:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1769076047.595992:0:20930:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: sanity-sec test_13: @@@@@@ FAIL: nodemap_add failed with 255 00010000:00010000:3.0:1769076047.664918:0:6173:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1769076047.983984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) b0abecde-06ba-4498-83fd-7513d38e1d30->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076047.983997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1769076047.984002:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076047.984013:0:8:0:(import.c:509:import_select_connection()) MGC192.168.204.117@tcp: connect to NID 192.168.204.117@tcp last attempt 1730 00000100:00080000:3.0:1769076047.984027:0:8:0:(import.c:588:import_select_connection()) MGC192.168.204.117@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076047.984035:0:8:0:(import.c:623:import_select_connection()) MGC192.168.204.117@tcp: import ffff895e4be1a800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000100:00100000:3.0:1769076047.984097:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5f80 x1855008961333120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076047.984144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076047.984149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1769076047.984152:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076047.984157:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1730 00000100:00080000:3.0:1769076047.984163:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1769076047.984169:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff895e42f9a800: import ffff895e42e8c000 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076047.984181:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076047.984202:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d4a80 x1855008961333248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1769076047.984217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 73a2201c-08e3-493e-ad2e-98b4627bffd5->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1769076047.984222:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1769076047.984225:0:8:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1769076047.984245:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: connect to NID 192.168.204.117@tcp last attempt 1730 00000100:00080000:3.0:1769076047.984253:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1769076047.984260:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961333120:192.168.204.117@tcp:250:kworker.0 00000100:00080000:3.0:1769076047.984270:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff895e42f9a800: import ffff895e42e89800 using connection 192.168.204.117@tcp/192.168.204.117@tcp 00000008:00100000:3.0:1769076047.984296:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681347 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1769076047.984323:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff895e453d5c00 x1855008961333376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1769076047.984424:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076047.984564:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076047/real 1769076047] req@ffff895e453d5f80 x1855008961333120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076047.984597:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5f80 x1855008961333120/t0(0) o250->MGC192.168.204.117@tcp@192.168.204.117@tcp:26/25 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076047.984612:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e4be1a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076047.984619:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076047.984626:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b0abecde-06ba-4498-83fd-7513d38e1d30:2402:1855008961333120:192.168.204.117@tcp:250:kworker.0 00000100:00100000:0.0:1769076047.984639:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961333248:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076047.984677:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076047.984704:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076047/real 1769076047] req@ffff895e453d4a80 x1855008961333248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076047.984722:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d4a80 x1855008961333248/t0(0) o38->lustre-MDT0000-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076047.984735:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e8c000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076047.984741:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076047.984799:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961333248:192.168.204.117@tcp:38:kworker.0 00000100:00100000:0.0:1769076047.984847:0:2402:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961333376:192.168.204.117@tcp:38:kworker.0 00000800:00000100:0.0:1769076047.984881:0:2402:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.204.117@tcp 00000100:00100000:0.0:1769076047.984897:0:2402:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1769076047/real 1769076047] req@ffff895e453d5c00 x1855008961333376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1769076047.984938:0:2402:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff895e453d5c00 x1855008961333376/t0(0) o38->lustre-MDT0001-mdc-ffff895e42f9a800@192.168.204.117@tcp:12/10 lens 520/544 e 0 to 1 dl 1769076072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1769076047.984949:0:2402:0:(import.c:65:import_set_state_nolock()) ffff895e42e89800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1769076047.984954:0:2402:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.204.117@tcp failed (-110) 00000100:00100000:0.0:1769076047.984958:0:2402:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff895e453d5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:73a2201c-08e3-493e-ad2e-98b4627bffd5:2402:1855008961333376:192.168.204.117@tcp:38:kworker.0 00010000:00010000:1.0:1769076048.687938:0:6653:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1769076049.711998:0:6172:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 394 lines, 394 kept, 0 dropped, 0 bad.